Input-to-Output Gate to Improve RNN Language Models
نویسندگان
چکیده
This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Output Gate (IOG)1. IOG has an extremely simple structure, and thus, can be easily combined with any RNN language models. Our experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG consistently boosts the performance of several different types of current topline RNN language models.
منابع مشابه
Neural Speed Reading via Skim-RNN
Inspired by the principles of speed reading, we introduce Skim-RNN, a recurrent neural network (RNN) that dynamically decides to update only a small fraction of the hidden state for relatively unimportant input tokens. Skim-RNN gives computational advantage over an RNN that always updates the entire hidden state. Skim-RNN uses the same input and output interfaces as a standard RNN and can be ea...
متن کاملEfficient Delay Characterization Method to Obtain the Output Waveform of Logic Gates Considering Glitches
Accurate delay calculation of circuit gates is very important in timing analysis of digital circuits. Waveform shapes on the input ports of logic gates should be considered, in the characterization phase of delay calculation, to obtain accurate gate delay values. Glitches and their temporal effect on circuit gate delays should be taken into account for this purpose. However, the explosive numbe...
متن کاملImproving Context Aware Language Models
Increased adaptability of RNN language models leads to improved predictions that benefit many applications. However, current methods do not take full advantage of the RNN structure. We show that the most widely-used approach to adaptation (concatenating the context with the word embedding at the input to the recurrent layer) is outperformed by a model that has some low-cost improvements: adapta...
متن کاملEfficient DMUs improvement based on input expenses reduction using data envelopment analysis
Network nowadays, the main purpose in the models designed by Data Envelopment Analysis (DEA), is to improve the outputs. In this method which is expressed by Khodabakhshi, with an output oriented BCC model, the output increases when the input increases. In this article we will discuss the efficient Decision Making Units (DMUs) in the input oriented BCC model to reduce the input expenses signifi...
متن کاملRecurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning
Sequence-to-sequence (Seq2seq) models have played an import role in the recent success of various natural language processing methods, such as machine translation, text summarization, and speech recognition. However, current Seq2seq models have trouble preserving global latent information from a long sequence of words. Variational autoencoder (VAE) alleviates this problem by learning a continuo...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2017